Skip to content

feat(compositor): CPU backend — WARP render + software decode, surfaced to the user - #162

Open
EtienneLescot wants to merge 206 commits into
release/v1.8.0from
feat/d3d-warp-fallback
Open

feat(compositor): CPU backend — WARP render + software decode, surfaced to the user#162
EtienneLescot wants to merge 206 commits into
release/v1.8.0from
feat/d3d-warp-fallback

Conversation

@EtienneLescot

@EtienneLescot EtienneLescot commented Jul 26, 2026

Copy link
Copy Markdown
Collaborator

This PR outgrew its title. It opened as a scoping PR: retry D3D11CreateDevice with D3D_DRIVER_TYPE_WARP when hardware init fails. Measurement refuted that fix (below). What shipped instead is a full CPU backend — WARP for rendering, libavcodec for decoding, system-memory frames for encoding — selected automatically and never silently.

Summary

Render, decode and encode are three independent axes, and no software rasteriser on any platform also decodes video (WARP here, lavapipe/SwiftShader on Linux, nothing at all on macOS). A fallback needs all three answered, not one driver-type swap:

axis hardware backend CPU backend
render D3D11 hardware WARP
decode D3D11VA (zero-copy) libavcodec software → swscale → NV12 upload (cpu_frames.rs)
encode h264_amf zero-copy h264_mf / libopenh264 via the existing ExportCodec::candidates()

The seam that keeps output identical: everything the compositor knows about a decoded frame is compositor::nv12_srvs() / compositor::tex_dims(), which read four fieldsdata[0] (NV12 texture), data[1] (array slice), width, height. Fill those and compositor.rs, every HLSL shader and the scene contract are untouched. A Metal or Vulkan port replaces d3d.rs + cpu_frames.rs and nothing else.

Why not the WARP retry this PR scoped

Measured on the reference machine (Ryzen 5 7520U + integrated Radeon, Windows 11):

probe result
WARP + D3D11_CREATE_DEVICE_VIDEO_SUPPORT DXGI_ERROR_UNSUPPORTED — device does not create
WARP without the flag creates at FL 11_1, but QueryInterface(ID3D11VideoDevice)E_NOINTERFACE, 0 decoder profiles
hardware, same call 20 decoder profiles

pipeline.rs hands the device to ffmpeg as the AVD3D11VADeviceContext, so preview and export decode every frame on it. A WARP device would produce zero frames — the retry would have converted a clear startup failure into an obscure ffmpeg one. Locked by crates/compositor/tests/warp_device_cannot_decode.rs: if that test ever fails, Windows gave WARP video capability and this design should be revisited.

The scoped questions, answered: WARP alone is acceptable for neither export nor preview; WGC capture stays hardware-only (see Known gaps); shader cost under WARP is fine except for two multi-tap loops (below).

What shipped

  • d3d.rsBackend::{Hardware, Cpu}; create (hardware-strict, for tests/goldens), create_auto (production: hardware → CPU fallback), cached probe(), and diagnose() which re-probes minus VIDEO_SUPPORT to separate "this adapter has no video decoder" (RDP / VM / Basic Render Driver — a driver update fixes it) from "no FL 11_1 adapter at all" (structural). That distinction is what the logs and the UI surface.
  • cpu_frames.rs (new) — software decode → swscale → NV12 → upload into one owned dynamic texture → present an AVFrame carrying the 4 seam fields.
  • pipeline.rsDecoder gains the CPU frame source; CPU export skips the encoder pool and VideoEncoder::send_composited() reads composed NV12 straight from the compositor (av_hwframe_transfer_data presupposes a D3D11 pool WARP cannot create — export died at enc hwdevice init before any encoder was tried).
  • live.rs / napi — the render thread's fatal error is stored and relayed as an Err on the next read_frame pull (~33 ms), instead of existing only as an eprintln! under a black canvas. Export uses create_auto.
  • Electron/TSprobeBackend() end to end (addon → service → IPC → useCompositorBackend() hook); the preview carries a small persistent CPU notice and the export dialog warns before the export starts. "none" (no addon at all: pure-web dev, jsdom) never warns — that is the normal state in development; only "cpu" is a degraded machine. Tested.
  • i18n — 13 locales: errors.previewCompositorUnavailable, cpuCompositor.{notice,exportWarning}.
  • Bench--backend hardware|cpu, --preview, --export; writes a PPM per run so a backend composing black can't post a flattering fps.

Measurements (frozen fixture, C1–C8)

  • Iso render — hardware vs CPU, same frame, all 6,220,800 channels: 93–95 % bit-identical, max deviation 3/255, mean levels matching (neither frame is blank). Every effect layer survives the backend swap; the residual is rasteriser/FP difference, so cross-backend goldens need a tolerance, not an exact baseline.
  • Preview — C1–C3 hold ~30 fps on WARP; C4+ sit at 6–9 fps. Two multi-tap sampling loops own the whole gap: background blur +4.53 ms hardware vs +75.43 ms WARP (17×), motion blur +2.30 vs +52.65 (23×). Everything else is within ~2.2× of the GPU.
  • Export (360 frames) — hardware 88.2 fps (h264_amf zero-copy); CPU backend 4.8 fps (h264_mf); CPU forced libopenh264 4.6 fps. All decode clean under ffmpeg -f null -.
  • ⚠️ Caveats: the absolutes need a re-run on a quiet machine (4 of 16 rows blew the <15 % spread gate with background processes live; the layer attribution is robust). And h264_mf winning on the CPU backend is a local artefact — Media Foundation picks its MFT independently of our device, so on a machine that has a GPU it still reaches hardware; OPENSCREEN_EXPORT_ENCODER=libopenh264 exercises the real last resort.

Product decisions

  1. No WARP-only retry — refuted by measurement.
  2. No effect degradation on the CPU path — ~8 fps is what the shipped 1.7.0 preview delivered; disabling blur/motion-blur would trade away the iso property. Slow and correct beats fast and wrong, in preview and export.
  3. Fallback is automatic but never silentcreate_auto + probeBackend() + notices.

Known gaps (not fixed here)

  • WGC capture fallback is unreachable on a host that fails D3D. is-native-windows-capture-available checks only the Windows build and helper-on-disk — never D3D. A host whose helper fails createD3DDevice gets available: true, commits to native, and startNativeWindowsRecordingIfAvailable rethrows instead of returning false, so the getDisplayMedia fallback is never reached. Not a one-line return false: the webcam preview stream has been deliberately released by then, so falling through would silently record screen-only. Fix = re-acquire the stream on the fallback route, or make the probe truthful (helper reports D3D capability before the renderer commits).
  • Preview effect policy deliberately absent — cfg.bg_blur / cfg.mblur_n are the levers if frame rate ever becomes the complaint.

Related issue

n/a — scoping PR that grew into the implementation.

Type of change

  • Feature
  • Performance

Release impact

  • Minor

Desktop impact

  • Windows

Testing

  • 1153 vitest + 81 Rust tests pass (--lib --tests; the 13 Rust doctest failures and the tsconfig.test.json errors are pre-existing on the base branch, verified by stashing).
  • crates/compositor/tests/warp_device_cannot_decode.rs pins the WARP probe results above.
  • Verified through the real addon (not mocks): probeBackend() returns "hardware" on a GPU machine and stays cached; CPU-backend export produces a file that decodes clean end to end.
  • Preview/export numbers from poc-d3d --cfg C1..C8 --backend {hardware,cpu} --frames 300 --repeat 3 on the frozen fixture (360 frames, 1920×1080 + 1920×1032, Constrained Baseline 60 fps); each run also writes a PPM frame for pixel inspection.

@coderabbitai

coderabbitai Bot commented Jul 26, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 95bdf0cd-d37b-4a79-a4c5-0a424be6b29f

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/d3d-warp-fallback

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

…s and snapping

- feat(ai-edition): multi-session chat history + multi-clip timeline with drag-drop
- feat(ai-edition): timeline interaction P0 — clip resize/delete/duplicate, region drag, Edit Clip dialog
- feat(ai-edition): timeline viewport spine — pan/zoom/reorder (T01-T09)
- feat(ai-edition): timeline lanes, navigator strip, Place-skip mode (T10-T18)
- feat(ai-edition): timeline polish — scrub preview, resequence, snap guides (T19-T25)
…atch the v4 design

- feat(ai-edition): refactor editor to OpenScreen Editor v4 (Media/Edit/Rec)
- style(ai-edition): flatten timeline region pills to v4 design
- style(ai-edition): match v4 timeline navigator to design
- style(ai-edition): align v4 timeline clip cards + ruler to design
- fix(ai-edition): hide timeline lane watermark when lane has regions
- style(ai-edition): float clip label as v4 blur chip over waveform
- docs(ai-edition): add v4 design-parity + import audit notes
@EtienneLescot
EtienneLescot marked this pull request as ready for review July 27, 2026 22:49
@EtienneLescot EtienneLescot changed the title feat(compositor): fall back to a WARP device when D3D11 hardware init fails feat(compositor): a CPU backend — WARP render + software decode, with the failure surfaced Jul 28, 2026
@EtienneLescot EtienneLescot changed the title feat(compositor): a CPU backend — WARP render + software decode, with the failure surfaced feat(compositor): CPU backend — WARP render + software decode, surfaced to the user Jul 28, 2026
EtienneLescot added a commit that referenced this pull request Jul 28, 2026
Empty commit so the scoping PR can exist; drop it when the real work starts.
Same move as #162 for the Windows CPU backend.
EtienneLescot added a commit that referenced this pull request Jul 28, 2026
Closes part of PR #183 (the scoping body). Defines the target architecture
(wgpu backend mirroring the D3D11 seam #162 introduced), the per-platform
frame contract, the WGSL shader port inventory (1:1 with shaders.hlsl's
9 entry points), the cosmic-text replacement for text.rs, the build/CI
plumbing for Linux, and a WP0..WP7 milestone plan. Adds a Linux port
subsection to native-compositor.md and a doc-table entry to README.md.

The spec body of PR #183 is updated separately to reference this doc.
EtienneLescot added a commit that referenced this pull request Jul 28, 2026
…d skeleton

Implements the foundation of PR #183.

WP0 — Cargo target gating
  - workspace deps: wgpu 24 (cross-platform, Vulkan/Metal/D3D12; pinned to
    the windows-0.58-compatible line so the existing D3D11 path stays on
    its validated toolchain), pollster for block_on.
  - compositor/Cargo.toml: wgpu + pollster deps cross-platform, the
    windows dep kept for the D3D11 path.
  - compositor-view-napi: the windows dep stays; lib.rs is split — the
    existing napi surface is cfg(windows)-gated into windows.rs, a
    non-Windows placeholder stands in until WP7 ships the Linux napi
    skeleton.
  - compositor/src/lib.rs: cfg(windows) on d3d/cpu_frames/text/compositor/
    live/pipeline (the D3D11 path); portable modules + new vk* stay ungated
    so the crate compiles everywhere.
  - poc-d3d: cfg(windows) on the Win32 body; non-Windows stub so
    `cargo check` on the target stays green.
  - compositor/build.rs: cross-build detection via cfg!(host_os) vs
    CARGO_CFG_TARGET_OS. When target != host, skip cc/bindgen and write
    a cached or stub ffi.rs so `cargo check --target x86_64-unknown-linux-gnu`
    from a Windows host passes without a cross-toolchain. Path separator
    made platform-aware.

WP1 — vk.rs (wgpu device + probe + lavapipe classification)
  - VkGpu { device, queue, backend, adapter_info } mirrors d3d::Gpu shape.
  - create (hardware-strict, for tests/goldens), create_blocking, create_auto
    (fallback HW -> lavapipe; full semantic equivalence with d3d::Backend
    in WP1+).
  - probe cached via OnceLock; returns the same "none" semantics as a
    missing addon on a machine without a Vulkan implementation.
  - classify() detects lavapipe by adapter name on Linux (llvmpipe /
    lavapipe substrings) and WARP on Windows — both routed to
    BackendKind::Cpu so probeBackend() reports "cpu" exactly as the
    Windows WARP rung does today.
  - Unit tests: probe-doesn't-panic, classify-hardware, classify-lavapipe.

WP2 — vk_frames.rs (NV12 upload skeleton)
  - VkFrameTex { y: r8unorm, uv: rg8unorm, width, height } mirrors the
    D3D11 NV12 texture that cpu_frames.rs presents today; the AVFrame
    carrier (data[0] = Box ptr) is the contract #162 introduced,
    generalised to carry the platform payload through crate boundaries.
  - VkFrames: AVFrame slot (av_frame_alloc), ensure_textures(w, h) creates
    the two R8/Rg8 textures when dimensions change.
  - Drop frees the AVFrame slot.
  - The swscale -> AV_PIX_FMT_NV12 -> queue.write_texture path lands in a
    follow-up WP2 commit; this commit establishes the seam and types.

vk_render.rs — module skeleton with VERSION constant and a test.
vk_shaders/blit.wgsl — minimal placeholder WGSL (fullscreen triangle +
black clear) for the include_str! source of the slice renderer.

Verified:
  - `cargo check --workspace`                                      green
  - `cargo check --workspace --target x86_64-unknown-linux-gnu`    green
    (uses OPENSCREEN_BINDGEN_CACHE_PATH to feed the bindgen output
    from a prior host build into the cross-build — see build.rs
    write_stub_ffi.)

WP3+ (slice renderer with WGSL core shaders and cross-backend golden
diff) is scoped in linux-compositor-port.md but lands in follow-up
commits to keep this PR reviewable.
EtienneLescot pushed a commit that referenced this pull request Jul 28, 2026
Branch off the CPU-backend seam established by PR #162 and split the
compositor crate into per-platform directories. Windows behaviour is
byte-identical (81 tests pass) and the macOS stubs that follow this
commit in the stack expose the same public surface (Backend, Gpu,
Compositor, Decoder, VideoEncoder, ExportCodec, ClipSource, LiveParams,
LayerCB...) and Err from every operation.

Renames + cfg dispatch:
  - crates/compositor/src/d3d_windows.rs           (from d3d.rs)
  - crates/compositor/src/cpu_frames_windows.rs    (from cpu_frames.rs)
  - crates/compositor/src/compositor_windows.rs    (from compositor.rs)
  - crates/compositor/src/pipeline_windows.rs      (from pipeline.rs)
  - crates/compositor/src/text_windows.rs          (from text.rs)
  - crates/compositor/wrapper_windows.h            (from wrapper.h)

lib.rs cfg-dispatches 'd3d', 'cpu_frames', 'compositor', 'text' across
the per-platform modules so call-sites stay portable.

live.rs: Win32 harness (run_standalone, host_proc, wide, client_size)
moved into a #[cfg(windows)] pub mod standalone_harness with a
#[cfg(target_os = "macos")] no-op stub for run_standalone (poc-d3d is
dev-only on Windows).

Cargo.toml: macOS deps (metal, objc, block, core-foundation) added
under [target.cfg(target_os = macos).dependencies]; windows dep
gated to cfg(windows). compositor-view-napi mirrors the gate.

crates/.cargo/config.toml: target-specific env block for macOS
(LIBCLANG_PATH not needed, MAC_FFMPEG_DIR falls back via build.rs).

build.rs: selects wrapper_windows.h vs wrapper_macos.h on
CARGO_CFG_TARGET_OS, sets bindgen --target=aarch64-apple-darwin +
-isysroot on macOS, finds MAC_FFMPEG_DIR or vendored
thirdparty/ffmpeg-n8.1.2-macos64-lgpl-shared.

Stacked on feat/d3d-warp-fallback (PR #162).
EtienneLescot pushed a commit that referenced this pull request Jul 29, 2026
"ISO with D3D" was an assertion. This makes the part of it that CAN be measured into
a failing test, and says plainly what the rest is worth.

The two backends cannot run on the same machine, so iso cannot be established by
comparing two rendered images. What can be — and what is the layer where divergence
actually happened on this branch — is the geometry: `plan_frame` is literally the same
code on both sides, so its 15 outputs must be bit-identical. `plan_frame_is_pinned_bit_for_bit`
pins them for a scene that exercises padding, crop, a displaced PiP camera, corner
radii and the zoom, comparing with `to_bits()` rather than an epsilon.

It runs in BOTH CI jobs — `rust-macos-compositor-check` and
`rust-windows-compositor-check`. If the two platforms ever compute different
placements, one of them goes red. That is the guarantee that was being claimed and
never held: `webcam_shape_code` returned "rounded" on Windows and "rectangle" on
macOS for months, and `LiveParams::default()` disagreed on two more fields, precisely
because nothing compared them.

What this test does NOT cover, and must not be read as covering: rasterisation. D3D11
and Metal will never be bit-identical — PR #162 measured 93-95 % of channels identical
with a max deviation of 3/255 between two backends on the SAME machine, and that is
the floor, not the target. Shader parity is held separately, by `shaders.metal` and
`shaders.hlsl` having been diffed line by line across all 14 modes; nine divergences
came out of that pass, three of them live.

The expected values are measured from this code, not chosen. A drift is a divergence
to explain, not a tolerance to loosen.
EtienneLescot added a commit that referenced this pull request Jul 30, 2026
PR #183 rebased from #162 base (old branch) to release/v1.8.0 (the
target the user asked for after conflict resolution on #162). The 41
extra commits between #162 and release/v1.8.0 are NOT in this PR — they
represent upstream work (native GIF export pipeline, pixi.js removal,
zoom/cursor fixes, etc.). The port reuses the same architecture and
files (vk.rs, vk_decode.rs, vk_frames.rs, vk_render.rs, text_cosmic.rs,
backend/{mod.rs,d3d11.rs,wgpu_backend.rs}, tests/vk_cross_golden.rs,
live_lin.rs, pipeline_lin.rs, compositor-view-napi/src/linux.rs) but on
top of the new base.

Key adaptations vs the old base:

- `compositor`/`d3d`/`live`/`pipeline`/`text`/`gif_export` are now
  `#[cfg(windows)]` modules in `compositor/src/lib.rs`. The Linux
  sibling modules (`vk*`, `backend`, `live_lin`, `pipeline_lin`,
  `text_cosmic`) are platform-agnostic (cross-compile via wgpu).
- `compositor::LayerCB` moved to `backend::LayerCB` (the platform-neutral
  definition matches HLSL `cbuffer Layer` and WGSL `struct Layer`
  byte-for-byte). Compositor's `LayerCB` is now `use crate::backend::LayerCB`.
- `compositor/src/compositor.rs` uses `use crate::backend::LayerCB`
  instead of defining its own. This unifies the two `LayerCB` symbols
  that previously caused a `E0308` mismatched-types error in d3d11.rs.
- `pipeline` (D3D11 arm) is NOT renamed to `pipeline_win`. The D3D11
  `run_composited`/`run_composited_multi` keep their original names and
  signatures. The Linux path (`pipeline_lin::Decoder`/`VideoEncoder`)
  is parallel.
- `compositor-view-napi`: split into `lib.rs` (cfg-gated `mod
  windows` / `mod linux` + `pub use`) plus `windows.rs` (full Windows
  addon, 622 lines) plus `linux.rs` (slice-based Linux addon, 230 lines).
  This replaces the previous "stub + Windows content in lib.rs" pattern
  the old branch had. The Windows dependency is `target.cfg(windows)`
  gated, so Linux builds don't pull `windows = "0.58"`.
- `compositor/build.rs` cross-build-aware: detects `cfg!(windows)` vs
  `CARGO_CFG_TARGET_OS` and either runs the full bindgen path (native
  host) or writes a cached/stub `ffi.rs` (cross-build, via
  `OPENSCREEN_BINDGEN_CACHE_PATH`). The wrapper.h includes
  `<libswscale/swscale.h>` (needed for `vk_frames.rs` swscale calls).
- `compositor/wrapper.h` includes `<libswscale/swscale.h>` for swscale
  bindings used by `vk_frames.rs`. The build.rs allowlists `sws_.*`.
- `cosmic-text = "0.19"` added as a workspace dep + compositor dep
  (the v0.19 features are default — there's no `fontdb`/`swash` feature
  flag in v0.19).
- `poc-d3d`: bench.rs's `export_gif` call updated to the new signature
  (`&[ClipSource]` instead of `&screen, &webcam, Some(&cursor)`).
  `gpu`/`comp` are now hoisted early so the GIF shortcut can run
  before the rest of the MP4-only setup. `mod app;` / `mod bench;`
  in `poc-d3d/src/main.rs` and the `windows` dep in its `Cargo.toml`
  are now `#[cfg(windows)]`/target-gated so Linux builds skip them.

What's in this branch (new files only):
- `crates/compositor/src/vk.rs` — wgpu device + lavapipe probe
  (`BackendKind::Cpu` when adapter is software via lavapipe on Mesa).
- `crates/compositor/src/vk_decode.rs` — `SwDecoder` (software ffmpeg
  decode via `avcodec`).
- `crates/compositor/src/vk_frames.rs` — `VkFrames` (NV12 split upload
  to two wgpu textures, AVFrame carrier).
- `crates/compositor/src/vk_render.rs` — `render_slice_c1` (slice
  renderer: WGSL layer.wgsl pipeline + bind groups + readback to PPM).
- `crates/compositor/src/vk_shaders/{layer,blur}.wgsl` — WGSL ports
  of the HLSL `ps_main` mode 0/1/2 + Kawase chain.
- `crates/compositor/src/text_cosmic.rs` — cosmic-text-based Linux
  text rasterizer (rustybuzz + swash + fontdb).
- `crates/compositor/src/backend/{mod,d3d11,wgpu_backend}.rs` —
  `CompBackend` trait + D3d11Backend forwarder (wraps existing
  Compositor) + WgpuBackend Linux impl.
- `crates/compositor/src/live_lin.rs` — Linux LiveView skeleton
  (cfg(not(windows))).
- `crates/compositor/src/pipeline_lin.rs` — Linux pipeline arm
  (Decoder / VideoEncoder / ExportCodec / run_composited with VAAPI
  candidates: `h264_vaapi` + `libopenh264`).
- `crates/compositor/tests/vk_cross_golden.rs` — slice integration test
  (decode frame 180 of the fixture, render, write PPM, log stats).
- `crates/compositor-view-napi/src/{linux.rs,windows.rs}` — split
  napi addon arms per platform.

Verified:
- `cargo check --workspace` (Windows host) — green.
- `cargo check --workspace --target x86_64-unknown-linux-gnu` (via
  `OPENSCREEN_BINDGEN_CACHE_PATH`) — green.
- `cargo test --tests -p openscreen-compositor` — 106 tests pass
  (100 unit + 2 export_timing + 3 output_geometry_golden +
  1 vk_cross_golden slice). The `vk_cross_golden` slice produces a
  1920×1080 RGBA8 PPM with mean RGB ≈ 232 on this Windows AMD iGPU.

What this branch deliberately does NOT include (WP4+–WP7 follow-ups
per `linux-compositor-port.md` §7):
- Full effect set on `WgpuBackend` (modes 4/6/7/8/9/10/11/12/13 —
  only modes 0/1/2 and blur are ported from HLSL today).
- `compose_frame` orchestration port to trait dispatch (the 800-line
  pure Rust math in `compositor.rs::compose_frame` still runs through
  the D3D11 path).
- Full Linux LiveView (multi-frame render thread) — `live_lin.rs` is
  a skeleton today.
- Linux MP4 muxer full implementation (ffmpeg avformat_new_stream +
  write_header + write_trailer) — `pipeline_lin::run_composited` is a
  skeleton that opens the encoder + runs the 1-frame path; the full
  muxer lands in WP7.

The architecture (`CompBackend` trait + `LayerCB` + cross-platform
`vk_*` modules + `vk_cross_golden` end-to-end test) is in place to
absorb each of these as small focused follow-up commits.
EtienneLescot added a commit that referenced this pull request Jul 30, 2026
…ing back to WARP

PR #162 scoped a WARP retry for the compositor's D3D11 device. Measured, WARP
cannot serve this pipeline at all -- and not for the expected reason (speed):

  - WARP + D3D11_CREATE_DEVICE_VIDEO_SUPPORT does not create. It returns
    DXGI_ERROR_UNSUPPORTED (0x887A0004).
  - Drop the flag and WARP creates at FL 11_1, but QueryInterface for
    ID3D11VideoDevice returns E_NOINTERFACE -- zero decoder profiles.

pipeline.rs hands Gpu's device to ffmpeg as the AVD3D11VADeviceContext, so
preview and export decode every frame on it. A WARP device would produce none:
the retry would only convert a clear startup failure into an obscure ffmpeg one.
crates/compositor/tests/warp_device_cannot_decode.rs pins both measurements and
fails if Windows ever changes them.

So there is no fallback, and the answer to "WARP for export or preview only?" is
neither. What the compositor does instead is fail in a way the user can act on:

  - Gpu::create re-probes on the failure path (same call, minus VIDEO_SUPPORT)
    to tell "this adapter has no video decoder" -- the RDP / VM case -- apart
    from "no FL 11_1 adapter at all", and says which, plus what to do. Export
    gets this for free: ExportDialog already renders the native message.
  - That message now reaches the preview too. create_view returns an id long
    before the render thread can die, so the failure existed only as an
    eprintln! and the user just saw a black canvas. The thread stores its fatal
    error in live::Shared, read_frame relays it as an Err on the next pull
    (~33 ms), and NativeCompositorOverlay renders it in place of the canvas.

The WGC capture helper stays hardware-only on purpose: it requests no
VIDEO_SUPPORT so a WARP device would be creatable there, but recording on a host
whose compositor cannot start only produces footage the user can neither edit
nor export.
EtienneLescot pushed a commit that referenced this pull request Jul 30, 2026
PR #162's CPU backend commit (13e1a31) was applied on top of the GIF
refactor that extracted `walk_composited_timeline` for the GIF path.
The conflict resolution put the WARP branching inside the timeline
walker, which doesn't have `out_w`/`out_h`/`out`/`params` -- those
are MP4-specific. This commit moves the encoder creation back into
`run_multi_inner` (where `out_w`/`out_h`/`out`/`params` are in scope)
and adds the `if software_frames { ... } else { ... }` branching in
the per-frame `on_frame` closure.

Also fixes the unclosed `run_gif_bench` delimiter from the conflict
resolution and adapts the call site to `export_gif`'s new signature
(`(clips, out_path, gpu, comp, cfg, params, progress)` -- the GIF
refactor moved from `(screen, webcam, cursor, out_path, params, dither)`
to the slice-2 MP4-shaped signature).
EtienneLescot pushed a commit that referenced this pull request Jul 30, 2026
Branch off the CPU-backend seam established by PR #162 and split the
compositor crate into per-platform directories. Windows behaviour is
byte-identical (81 tests pass) and the macOS stubs that follow this
commit in the stack expose the same public surface (Backend, Gpu,
Compositor, Decoder, VideoEncoder, ExportCodec, ClipSource, LiveParams,
LayerCB...) and Err from every operation.

Renames + cfg dispatch:
  - crates/compositor/src/d3d_windows.rs           (from d3d.rs)
  - crates/compositor/src/cpu_frames_windows.rs    (from cpu_frames.rs)
  - crates/compositor/src/compositor_windows.rs    (from compositor.rs)
  - crates/compositor/src/pipeline_windows.rs      (from pipeline.rs)
  - crates/compositor/src/text_windows.rs          (from text.rs)
  - crates/compositor/wrapper_windows.h            (from wrapper.h)

lib.rs cfg-dispatches 'd3d', 'cpu_frames', 'compositor', 'text' across
the per-platform modules so call-sites stay portable.

live.rs: Win32 harness (run_standalone, host_proc, wide, client_size)
moved into a #[cfg(windows)] pub mod standalone_harness with a
dev-only on Windows).

Cargo.toml: macOS deps (metal, objc, block, core-foundation) added
under [target.cfg(target_os = macos).dependencies]; windows dep
gated to cfg(windows). compositor-view-napi mirrors the gate.

crates/.cargo/config.toml: target-specific env block for macOS
(LIBCLANG_PATH not needed, MAC_FFMPEG_DIR falls back via build.rs).

build.rs: selects wrapper_windows.h vs wrapper_macos.h on
CARGO_CFG_TARGET_OS, sets bindgen --target=aarch64-apple-darwin +
-isysroot on macOS, finds MAC_FFMPEG_DIR or vendored
thirdparty/ffmpeg-n8.1.2-macos64-lgpl-shared.

Stacked on feat/d3d-warp-fallback (PR #162).
EtienneLescot pushed a commit that referenced this pull request Jul 30, 2026
"ISO with D3D" was an assertion. This makes the part of it that CAN be measured into
a failing test, and says plainly what the rest is worth.

The two backends cannot run on the same machine, so iso cannot be established by
comparing two rendered images. What can be — and what is the layer where divergence
actually happened on this branch — is the geometry: `plan_frame` is literally the same
code on both sides, so its 15 outputs must be bit-identical. `plan_frame_is_pinned_bit_for_bit`
pins them for a scene that exercises padding, crop, a displaced PiP camera, corner
radii and the zoom, comparing with `to_bits()` rather than an epsilon.

It runs in BOTH CI jobs — `rust-macos-compositor-check` and
`rust-windows-compositor-check`. If the two platforms ever compute different
placements, one of them goes red. That is the guarantee that was being claimed and
never held: `webcam_shape_code` returned "rounded" on Windows and "rectangle" on
macOS for months, and `LiveParams::default()` disagreed on two more fields, precisely
because nothing compared them.

What this test does NOT cover, and must not be read as covering: rasterisation. D3D11
and Metal will never be bit-identical — PR #162 measured 93-95 % of channels identical
with a max deviation of 3/255 between two backends on the SAME machine, and that is
the floor, not the target. Shader parity is held separately, by `shaders.metal` and
`shaders.hlsl` having been diffed line by line across all 14 modes; nine divergences
came out of that pass, three of them live.

The expected values are measured from this code, not chosen. A drift is a divergence
to explain, not a tolerance to loosen.
EtienneLescot added a commit that referenced this pull request Jul 30, 2026
…ing back to WARP

PR #162 scoped a WARP retry for the compositor's D3D11 device. Measured, WARP
cannot serve this pipeline at all -- and not for the expected reason (speed):

  - WARP + D3D11_CREATE_DEVICE_VIDEO_SUPPORT does not create. It returns
    DXGI_ERROR_UNSUPPORTED (0x887A0004).
  - Drop the flag and WARP creates at FL 11_1, but QueryInterface for
    ID3D11VideoDevice returns E_NOINTERFACE -- zero decoder profiles.

pipeline.rs hands Gpu's device to ffmpeg as the AVD3D11VADeviceContext, so
preview and export decode every frame on it. A WARP device would produce none:
the retry would only convert a clear startup failure into an obscure ffmpeg one.
crates/compositor/tests/warp_device_cannot_decode.rs pins both measurements and
fails if Windows ever changes them.

So there is no fallback, and the answer to "WARP for export or preview only?" is
neither. What the compositor does instead is fail in a way the user can act on:

  - Gpu::create re-probes on the failure path (same call, minus VIDEO_SUPPORT)
    to tell "this adapter has no video decoder" -- the RDP / VM case -- apart
    from "no FL 11_1 adapter at all", and says which, plus what to do. Export
    gets this for free: ExportDialog already renders the native message.
  - That message now reaches the preview too. create_view returns an id long
    before the render thread can die, so the failure existed only as an
    eprintln! and the user just saw a black canvas. The thread stores its fatal
    error in live::Shared, read_frame relays it as an Err on the next pull
    (~33 ms), and NativeCompositorOverlay renders it in place of the canvas.

The WGC capture helper stays hardware-only on purpose: it requests no
VIDEO_SUPPORT so a WARP device would be creatable there, but recording on a host
whose compositor cannot start only produces footage the user can neither edit
nor export.
EtienneLescot pushed a commit that referenced this pull request Jul 30, 2026
PR #162's CPU backend commit (13e1a31) was applied on top of the GIF
refactor that extracted `walk_composited_timeline` for the GIF path.
The conflict resolution put the WARP branching inside the timeline
walker, which doesn't have `out_w`/`out_h`/`out`/`params` -- those
are MP4-specific. This commit moves the encoder creation back into
`run_multi_inner` (where `out_w`/`out_h`/`out`/`params` are in scope)
and adds the `if software_frames { ... } else { ... }` branching in
the per-frame `on_frame` closure.

Also fixes the unclosed `run_gif_bench` delimiter from the conflict
resolution and adapts the call site to `export_gif`'s new signature
(`(clips, out_path, gpu, comp, cfg, params, progress)` -- the GIF
refactor moved from `(screen, webcam, cursor, out_path, params, dither)`
to the slice-2 MP4-shaped signature).
EtienneLescot pushed a commit that referenced this pull request Jul 30, 2026
Branch off the CPU-backend seam established by PR #162 and split the
compositor crate into per-platform directories. Windows behaviour is
byte-identical (81 tests pass) and the macOS stubs that follow this
commit in the stack expose the same public surface (Backend, Gpu,
Compositor, Decoder, VideoEncoder, ExportCodec, ClipSource, LiveParams,
LayerCB...) and Err from every operation.

Renames + cfg dispatch:
  - crates/compositor/src/d3d_windows.rs           (from d3d.rs)
  - crates/compositor/src/cpu_frames_windows.rs    (from cpu_frames.rs)
  - crates/compositor/src/compositor_windows.rs    (from compositor.rs)
  - crates/compositor/src/pipeline_windows.rs      (from pipeline.rs)
  - crates/compositor/src/text_windows.rs          (from text.rs)
  - crates/compositor/wrapper_windows.h            (from wrapper.h)

lib.rs cfg-dispatches 'd3d', 'cpu_frames', 'compositor', 'text' across
the per-platform modules so call-sites stay portable.

live.rs: Win32 harness (run_standalone, host_proc, wide, client_size)
moved into a #[cfg(windows)] pub mod standalone_harness with a
dev-only on Windows).

Cargo.toml: macOS deps (metal, objc, block, core-foundation) added
under [target.cfg(target_os = macos).dependencies]; windows dep
gated to cfg(windows). compositor-view-napi mirrors the gate.

crates/.cargo/config.toml: target-specific env block for macOS
(LIBCLANG_PATH not needed, MAC_FFMPEG_DIR falls back via build.rs).

build.rs: selects wrapper_windows.h vs wrapper_macos.h on
CARGO_CFG_TARGET_OS, sets bindgen --target=aarch64-apple-darwin +
-isysroot on macOS, finds MAC_FFMPEG_DIR or vendored
thirdparty/ffmpeg-n8.1.2-macos64-lgpl-shared.

Stacked on feat/d3d-warp-fallback (PR #162).
EtienneLescot pushed a commit that referenced this pull request Jul 30, 2026
"ISO with D3D" was an assertion. This makes the part of it that CAN be measured into
a failing test, and says plainly what the rest is worth.

The two backends cannot run on the same machine, so iso cannot be established by
comparing two rendered images. What can be — and what is the layer where divergence
actually happened on this branch — is the geometry: `plan_frame` is literally the same
code on both sides, so its 15 outputs must be bit-identical. `plan_frame_is_pinned_bit_for_bit`
pins them for a scene that exercises padding, crop, a displaced PiP camera, corner
radii and the zoom, comparing with `to_bits()` rather than an epsilon.

It runs in BOTH CI jobs — `rust-macos-compositor-check` and
`rust-windows-compositor-check`. If the two platforms ever compute different
placements, one of them goes red. That is the guarantee that was being claimed and
never held: `webcam_shape_code` returned "rounded" on Windows and "rectangle" on
macOS for months, and `LiveParams::default()` disagreed on two more fields, precisely
because nothing compared them.

What this test does NOT cover, and must not be read as covering: rasterisation. D3D11
and Metal will never be bit-identical — PR #162 measured 93-95 % of channels identical
with a max deviation of 3/255 between two backends on the SAME machine, and that is
the floor, not the target. Shader parity is held separately, by `shaders.metal` and
`shaders.hlsl` having been diffed line by line across all 14 modes; nine divergences
came out of that pass, three of them live.

The expected values are measured from this code, not chosen. A drift is a divergence
to explain, not a tolerance to loosen.
EtienneLescot added a commit that referenced this pull request Jul 31, 2026
…ing back to WARP

PR #162 scoped a WARP retry for the compositor's D3D11 device. Measured, WARP
cannot serve this pipeline at all -- and not for the expected reason (speed):

  - WARP + D3D11_CREATE_DEVICE_VIDEO_SUPPORT does not create. It returns
    DXGI_ERROR_UNSUPPORTED (0x887A0004).
  - Drop the flag and WARP creates at FL 11_1, but QueryInterface for
    ID3D11VideoDevice returns E_NOINTERFACE -- zero decoder profiles.

pipeline.rs hands Gpu's device to ffmpeg as the AVD3D11VADeviceContext, so
preview and export decode every frame on it. A WARP device would produce none:
the retry would only convert a clear startup failure into an obscure ffmpeg one.
crates/compositor/tests/warp_device_cannot_decode.rs pins both measurements and
fails if Windows ever changes them.

So there is no fallback, and the answer to "WARP for export or preview only?" is
neither. What the compositor does instead is fail in a way the user can act on:

  - Gpu::create re-probes on the failure path (same call, minus VIDEO_SUPPORT)
    to tell "this adapter has no video decoder" -- the RDP / VM case -- apart
    from "no FL 11_1 adapter at all", and says which, plus what to do. Export
    gets this for free: ExportDialog already renders the native message.
  - That message now reaches the preview too. create_view returns an id long
    before the render thread can die, so the failure existed only as an
    eprintln! and the user just saw a black canvas. The thread stores its fatal
    error in live::Shared, read_frame relays it as an Err on the next pull
    (~33 ms), and NativeCompositorOverlay renders it in place of the canvas.

The WGC capture helper stays hardware-only on purpose: it requests no
VIDEO_SUPPORT so a WARP device would be creatable there, but recording on a host
whose compositor cannot start only produces footage the user can neither edit
nor export.
EtienneLescot added a commit that referenced this pull request Jul 31, 2026
PR #162's CPU backend commit (13e1a31) was applied on top of the GIF
refactor that extracted `walk_composited_timeline` for the GIF path.
The conflict resolution put the WARP branching inside the timeline
walker, which doesn't have `out_w`/`out_h`/`out`/`params` -- those
are MP4-specific. This commit moves the encoder creation back into
`run_multi_inner` (where `out_w`/`out_h`/`out`/`params` are in scope)
and adds the `if software_frames { ... } else { ... }` branching in
the per-frame `on_frame` closure.

Also fixes the unclosed `run_gif_bench` delimiter from the conflict
resolution and adapts the call site to `export_gif`'s new signature
(`(clips, out_path, gpu, comp, cfg, params, progress)` -- the GIF
refactor moved from `(screen, webcam, cursor, out_path, params, dither)`
to the slice-2 MP4-shaped signature).
EtienneLescot added a commit that referenced this pull request Jul 31, 2026
Branch off the CPU-backend seam established by PR #162 and split the
compositor crate into per-platform directories. Windows behaviour is
byte-identical (81 tests pass) and the macOS stubs that follow this
commit in the stack expose the same public surface (Backend, Gpu,
Compositor, Decoder, VideoEncoder, ExportCodec, ClipSource, LiveParams,
LayerCB...) and Err from every operation.

Renames + cfg dispatch:
  - crates/compositor/src/d3d_windows.rs           (from d3d.rs)
  - crates/compositor/src/cpu_frames_windows.rs    (from cpu_frames.rs)
  - crates/compositor/src/compositor_windows.rs    (from compositor.rs)
  - crates/compositor/src/pipeline_windows.rs      (from pipeline.rs)
  - crates/compositor/src/text_windows.rs          (from text.rs)
  - crates/compositor/wrapper_windows.h            (from wrapper.h)

lib.rs cfg-dispatches 'd3d', 'cpu_frames', 'compositor', 'text' across
the per-platform modules so call-sites stay portable.

live.rs: Win32 harness (run_standalone, host_proc, wide, client_size)
moved into a #[cfg(windows)] pub mod standalone_harness with a
dev-only on Windows).

Cargo.toml: macOS deps (metal, objc, block, core-foundation) added
under [target.cfg(target_os = macos).dependencies]; windows dep
gated to cfg(windows). compositor-view-napi mirrors the gate.

crates/.cargo/config.toml: target-specific env block for macOS
(LIBCLANG_PATH not needed, MAC_FFMPEG_DIR falls back via build.rs).

build.rs: selects wrapper_windows.h vs wrapper_macos.h on
CARGO_CFG_TARGET_OS, sets bindgen --target=aarch64-apple-darwin +
-isysroot on macOS, finds MAC_FFMPEG_DIR or vendored
thirdparty/ffmpeg-n8.1.2-macos64-lgpl-shared.

Stacked on feat/d3d-warp-fallback (PR #162).
EtienneLescot added a commit that referenced this pull request Jul 31, 2026
"ISO with D3D" was an assertion. This makes the part of it that CAN be measured into
a failing test, and says plainly what the rest is worth.

The two backends cannot run on the same machine, so iso cannot be established by
comparing two rendered images. What can be — and what is the layer where divergence
actually happened on this branch — is the geometry: `plan_frame` is literally the same
code on both sides, so its 15 outputs must be bit-identical. `plan_frame_is_pinned_bit_for_bit`
pins them for a scene that exercises padding, crop, a displaced PiP camera, corner
radii and the zoom, comparing with `to_bits()` rather than an epsilon.

It runs in BOTH CI jobs — `rust-macos-compositor-check` and
`rust-windows-compositor-check`. If the two platforms ever compute different
placements, one of them goes red. That is the guarantee that was being claimed and
never held: `webcam_shape_code` returned "rounded" on Windows and "rectangle" on
macOS for months, and `LiveParams::default()` disagreed on two more fields, precisely
because nothing compared them.

What this test does NOT cover, and must not be read as covering: rasterisation. D3D11
and Metal will never be bit-identical — PR #162 measured 93-95 % of channels identical
with a max deviation of 3/255 between two backends on the SAME machine, and that is
the floor, not the target. Shader parity is held separately, by `shaders.metal` and
`shaders.hlsl` having been diffed line by line across all 14 modes; nine divergences
came out of that pass, three of them live.

The expected values are measured from this code, not chosen. A drift is a divergence
to explain, not a tolerance to loosen.
EtienneLescot added a commit that referenced this pull request Aug 1, 2026
…ing back to WARP

PR #162 scoped a WARP retry for the compositor's D3D11 device. Measured, WARP
cannot serve this pipeline at all -- and not for the expected reason (speed):

  - WARP + D3D11_CREATE_DEVICE_VIDEO_SUPPORT does not create. It returns
    DXGI_ERROR_UNSUPPORTED (0x887A0004).
  - Drop the flag and WARP creates at FL 11_1, but QueryInterface for
    ID3D11VideoDevice returns E_NOINTERFACE -- zero decoder profiles.

pipeline.rs hands Gpu's device to ffmpeg as the AVD3D11VADeviceContext, so
preview and export decode every frame on it. A WARP device would produce none:
the retry would only convert a clear startup failure into an obscure ffmpeg one.
crates/compositor/tests/warp_device_cannot_decode.rs pins both measurements and
fails if Windows ever changes them.

So there is no fallback, and the answer to "WARP for export or preview only?" is
neither. What the compositor does instead is fail in a way the user can act on:

  - Gpu::create re-probes on the failure path (same call, minus VIDEO_SUPPORT)
    to tell "this adapter has no video decoder" -- the RDP / VM case -- apart
    from "no FL 11_1 adapter at all", and says which, plus what to do. Export
    gets this for free: ExportDialog already renders the native message.
  - That message now reaches the preview too. create_view returns an id long
    before the render thread can die, so the failure existed only as an
    eprintln! and the user just saw a black canvas. The thread stores its fatal
    error in live::Shared, read_frame relays it as an Err on the next pull
    (~33 ms), and NativeCompositorOverlay renders it in place of the canvas.

The WGC capture helper stays hardware-only on purpose: it requests no
VIDEO_SUPPORT so a WARP device would be creatable there, but recording on a host
whose compositor cannot start only produces footage the user can neither edit
nor export.
EtienneLescot added a commit that referenced this pull request Aug 1, 2026
PR #162's CPU backend commit (13e1a31) was applied on top of the GIF
refactor that extracted `walk_composited_timeline` for the GIF path.
The conflict resolution put the WARP branching inside the timeline
walker, which doesn't have `out_w`/`out_h`/`out`/`params` -- those
are MP4-specific. This commit moves the encoder creation back into
`run_multi_inner` (where `out_w`/`out_h`/`out`/`params` are in scope)
and adds the `if software_frames { ... } else { ... }` branching in
the per-frame `on_frame` closure.

Also fixes the unclosed `run_gif_bench` delimiter from the conflict
resolution and adapts the call site to `export_gif`'s new signature
(`(clips, out_path, gpu, comp, cfg, params, progress)` -- the GIF
refactor moved from `(screen, webcam, cursor, out_path, params, dither)`
to the slice-2 MP4-shaped signature).
EtienneLescot added a commit that referenced this pull request Aug 1, 2026
Branch off the CPU-backend seam established by PR #162 and split the
compositor crate into per-platform directories. Windows behaviour is
byte-identical (81 tests pass) and the macOS stubs that follow this
commit in the stack expose the same public surface (Backend, Gpu,
Compositor, Decoder, VideoEncoder, ExportCodec, ClipSource, LiveParams,
LayerCB...) and Err from every operation.

Renames + cfg dispatch:
  - crates/compositor/src/d3d_windows.rs           (from d3d.rs)
  - crates/compositor/src/cpu_frames_windows.rs    (from cpu_frames.rs)
  - crates/compositor/src/compositor_windows.rs    (from compositor.rs)
  - crates/compositor/src/pipeline_windows.rs      (from pipeline.rs)
  - crates/compositor/src/text_windows.rs          (from text.rs)
  - crates/compositor/wrapper_windows.h            (from wrapper.h)

lib.rs cfg-dispatches 'd3d', 'cpu_frames', 'compositor', 'text' across
the per-platform modules so call-sites stay portable.

live.rs: Win32 harness (run_standalone, host_proc, wide, client_size)
moved into a #[cfg(windows)] pub mod standalone_harness with a
dev-only on Windows).

Cargo.toml: macOS deps (metal, objc, block, core-foundation) added
under [target.cfg(target_os = macos).dependencies]; windows dep
gated to cfg(windows). compositor-view-napi mirrors the gate.

crates/.cargo/config.toml: target-specific env block for macOS
(LIBCLANG_PATH not needed, MAC_FFMPEG_DIR falls back via build.rs).

build.rs: selects wrapper_windows.h vs wrapper_macos.h on
CARGO_CFG_TARGET_OS, sets bindgen --target=aarch64-apple-darwin +
-isysroot on macOS, finds MAC_FFMPEG_DIR or vendored
thirdparty/ffmpeg-n8.1.2-macos64-lgpl-shared.

Stacked on feat/d3d-warp-fallback (PR #162).
EtienneLescot added a commit that referenced this pull request Aug 1, 2026
"ISO with D3D" was an assertion. This makes the part of it that CAN be measured into
a failing test, and says plainly what the rest is worth.

The two backends cannot run on the same machine, so iso cannot be established by
comparing two rendered images. What can be — and what is the layer where divergence
actually happened on this branch — is the geometry: `plan_frame` is literally the same
code on both sides, so its 15 outputs must be bit-identical. `plan_frame_is_pinned_bit_for_bit`
pins them for a scene that exercises padding, crop, a displaced PiP camera, corner
radii and the zoom, comparing with `to_bits()` rather than an epsilon.

It runs in BOTH CI jobs — `rust-macos-compositor-check` and
`rust-windows-compositor-check`. If the two platforms ever compute different
placements, one of them goes red. That is the guarantee that was being claimed and
never held: `webcam_shape_code` returned "rounded" on Windows and "rectangle" on
macOS for months, and `LiveParams::default()` disagreed on two more fields, precisely
because nothing compared them.

What this test does NOT cover, and must not be read as covering: rasterisation. D3D11
and Metal will never be bit-identical — PR #162 measured 93-95 % of channels identical
with a max deviation of 3/255 between two backends on the SAME machine, and that is
the floor, not the target. Shader parity is held separately, by `shaders.metal` and
`shaders.hlsl` having been diffed line by line across all 14 modes; nine divergences
came out of that pass, three of them live.

The expected values are measured from this code, not chosen. A drift is a divergence
to explain, not a tolerance to loosen.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant